Workflow engine

• https://kestra.io/

• https://github.com/kestra-io/helm-charts

• ArtifactHUB kestra

• https://hub.docker.com/r/kestra/kestra kestra/kestra
  • Kestra image tags here
  • Add version to helm chart

• Reserve k8s resources in yml for base services: kestra, …

• kestra: capture logging to send back on failure.

• kestra: read report and tasks from file

• kestra: Use variables / secrets in flow.
  • Have to purchase enterprise?

• Code

  • https://github.com/kestra-io/kestra

  • https://github.com/kestra-io/helm-charts

  • https://github.com/kestra-io/terraform-deployments

  • https://github.com/kestra-io/plugin-aws

• Notes

  • Pebble templating

  • Kestra not in standalone mod takes about 8G of memory

  • Adjust the multipart.threshold and multipart.chunksize in the Kestra configuration to optimize the upload process.

• Doc Links

  • Getting Started

  • OpenSource API Ref

  • Plugin List

  • Plugin Defaults

  • JSON Filters

  • AWS Batch Tasks

  • Storing flows in GIT

  • Trigger from the outside using webhook

  • Example of run sub-flow in batches

  • Secrets

    • echo -n val | base64

  • Http Requests

    • Mozilla HTTP docs

    • Download a file

  • Expressions

    • Secret

    • Pebble

      • Pebble Templating Engine

      • pebbletemplates.io

      • function list
        • read
        • jq manual

    • Function

    • Flow Control

  • Outputs

    • Conponent Outputs

  • Serdes - Serialization

    • https://github.com/kestra-io/plugin-serdes

    • JSON writer

  • Storage

    • Backup and Restore

    • Dev Guide Storage

    • Arch Internal Storage

    • Minio S3 like for local dev

      • See minio section on S3 page

      S3

  • Configuration

    • Server config

  • Restack Kestra Docs

    • S3 Integration — does it work with k8s?

• Using API

  • https://kestra.io/docs/how-to-guides/synchonous-executions-api

  echo -n "username:password" | base64 | pbcopy # in x-flux
  
  export BASE='https://kestra.k1.int.x.com'
  export AUTH='Authorization: Basic xxx'
  
  curl -X POST \
  'https://localhost:8080/api/v1/executions/company.team/myflow?wait=true' \
  -H 'Authorization: Basic <encoded-string>'
  
  curl -X GET \
  "$BASE/api/v1/executions/?namespace=rufas&flowId=rufasflow" \
  -H $AUTH -H "Content-Type: application/json"
  
  curl -X DELETE \
  "$BASE/api/v1/executions/by-query?namespace=dev&flowId=test" \
  -H $AUTH -H "Content-Type: application/json"
  

• Terraform

  • Using terraform with ECS and Batch

• Things to learn

  • Security

    • UI, API, …

  • How to keep it backed up?

    • flows

    • db

    • config - captured in flux?

  • How to use GIT or S3 as storage to track flows

    • Think I have this working

  • Slack is pretty responsive to questions

    • https://app.slack.com/client/T01JX6XH5KN/C03F317FP9V

    • myron.overton@gmail.com

  • Secrets config

    • https://kestra.io/docs/configuration-guide/encryption

    # add to helm release
      values:
        secrets:
          kestra: 
            encryption: 
              secret-key: xxxx
    

# render expression
>{{outputs}}<

"kestra.api.url": "https://api.kestra.io/api/v1",
"kestra.schema.match-path": ""

• kestra: New encryption key. https://kestra.io/docs/configuration-guide/encryption

• Kestra Deploy to Prod

  • ONLY one at a time

  • Add cron

    */5 * * * * cd $HOME/deploy && ./bin/exec-console.sh x:rufas:sync > ./log/cron/rufas-sync.log 2>&1
    
    {{ jq outputs.counts.results '. | select(. != null) | .[]' }}

  • deploy backend, eval-app, forms

  • Todo

    • Add herdNum limit to kestra flow

    • Fail if herdNum > x

    • kestra: add herdNum limit to stop large runs

      • See logic from William

      • Terraform the batch envs for the diff sizes

      • Look at python script to manage aws batch in a kestra step

  • Kestra to 128G instance $1300/mo

    • Changed to 64G on 9/27

  We ask them to scale down next week. 
  Even if it’s 4000 cows, run 400, and just scale 
  the milk, calf and energy inputs associated.

  • Flow result back to users

    • what is the size for input,output,logs.

    • Generate S3 urls for those

    • rufas: Need to send status to Slack

  ---